Scoping Meta-Review of Methods Used to Assess Artificial Intelligence-Based Medical Devices for Heart Failure

Artificial intelligence and machine learning (AI/ML) are playing increasingly important roles, permeating the field of medical devices (MDs). This rapid progress has not yet been matched by the Health Technology Assessment (HTA) process, which still needs to define a common methodology for assessing AI/ML-based MDs. To collect existing evidence from the literature about the methods used to assess AI-based MDs, with a specific focus on those used for the management of heart failure (HF), the International Federation of Medical and Biological Engineering (IFMBE) conducted a scoping meta-review. This manuscript presents the results of this search, which covered the period from January 1974 to October 2022. After careful independent screening, 21 reviews, mainly conducted in North America and Europe, were retained and included. Among the findings were that deep learning is the most commonly utilised method and that electronic health records and registries are among the most prevalent sources of data for AI/ML algorithms. Out of the 21 included reviews, 19 focused on risk prediction and/or the early diagnosis of HF. Furthermore, 10 reviews provided evidence of the impact on the incidence/progression of HF, and 13 on the length of stay. From an HTA perspective, the main areas requiring improvement are the quality assessment of studies on AI/ML (included in 11 out of 21 reviews) and their data sources, as well as the definition of the criteria used to assess the selection of the most appropriate AI/ML algorithm. © 2023 by the authors

Many quality measurements, but few quality measures assessing the quality of breast cancer care in women: A systematic review

BACKGROUND: Breast cancer in women is increasingly frequent, and care is complex, onerous and expensive, all of which lend urgency to improvements in care. Quality measurement is essential to monitor effectiveness and to guide improvements in healthcare. METHODS: Ten databases, including Medline, were searched electronically to identify measures assessing the quality of breast cancer care in women (diagnosis, treatment, followup, documentation of care). Eligible studies measured adherence to standards of breast cancer care in women diagnosed with, or in treatment for, any histological type of adenocarcinoma of the breast. Reference lists of studies, review articles, web sites, and files of experts were searched manually. Evidence appraisal entailed dual independent assessments of data (e.g., indicators used in quality measurement). The extent of each quality indicator's scientific validation as a measure was assessed. The American Society of Clinical Oncology (ASCO) was asked to contribute quality measures under development. RESULTS: Sixty relevant reports identified 58 studies with 143 indicators assessing adherence to quality breast cancer care. A paucity of validated indicators (n = 12), most of which assessed quality of life, only permitted a qualitative data synthesis. Most quality indicators evaluated processes of care. CONCLUSION: While some studies revealed patterns of under-use of care, all adherence data require confirmation using validated quality measures. ASCO's current development of a set of quality measures relating to breast cancer care may hold the key to conducting definitive studies

An alternative to the hand searching gold standard: validating methodological search filters using relative recall

BACKGROUND: Search filters or hedges play an important role in evidence-based medicine but their development depends on the availability of a "gold standard" – a reference standard against which to establish the performance of the filter. We demonstrate the feasibility of using relative recall of included studies from multiple systematic reviews to validate methodological search filters as an alternative to validation against a gold standard formed through hand searching. METHODS: We identified 105 Cochrane reviews that used the Highly Sensitive Search Strategy (HSSS), included randomized or quasi-randomized controlled trials, and reported their included studies. We measured the ability of two published and one novel variant of the HSSS to retrieve the MEDLINE-index studies included in these reviews. RESULTS: The systematic reviews were comprehensive in their searches. 72% of included primary studies were indexed in MEDLINE. Relative recall of the three strategies ranged from .98 to .91 across all reviews and more comprehensive strategies showed higher recall. CONCLUSION: An approach using relative recall instead of a hand searching gold standard proved feasible and produced recall figures that were congruent with previously published figures for the HSSS. This technique would permit validation of a methodological filter using a collection of approximately 100 studies of the chosen design drawn from the included studies of multiple systematic reviews that used comprehensive search strategies

Can electronic search engines optimize screening of search results in systematic reviews: an empirical study

BACKGROUND: Most electronic search efforts directed at identifying primary studies for inclusion in systematic reviews rely on the optimal Boolean search features of search interfaces such as DIALOG(® )and Ovid™. Our objective is to test the ability of an Ultraseek(® )search engine to rank MEDLINE(® )records of the included studies of Cochrane reviews within the top half of all the records retrieved by the Boolean MEDLINE search used by the reviewers. METHODS: Collections were created using the MEDLINE bibliographic records of included and excluded studies listed in the review and all records retrieved by the MEDLINE search. Records were converted to individual HTML files. Collections of records were indexed and searched through a statistical search engine, Ultraseek, using review-specific search terms. Our data sources, systematic reviews published in the Cochrane library, were included if they reported using at least one phase of the Cochrane Highly Sensitive Search Strategy (HSSS), provided citations for both included and excluded studies and conducted a meta-analysis using a binary outcome measure. Reviews were selected if they yielded between 1000–6000 records when the MEDLINE search strategy was replicated. RESULTS: Nine Cochrane reviews were included. Included studies within the Cochrane reviews were found within the first 500 retrieved studies more often than would be expected by chance. Across all reviews, recall of included studies into the top 500 was 0.70. There was no statistically significant difference in ranking when comparing included studies with just the subset of excluded studies listed as excluded in the published review. CONCLUSION: The relevance ranking provided by the search engine was better than expected by chance and shows promise for the preliminary evaluation of large results from Boolean searches. A statistical search engine does not appear to be able to make fine discriminations concerning the relevance of bibliographic records that have been pre-screened by systematic reviewers

Global, regional, and national incidence, prevalence, and years lived with disability for 354 diseases and injuries for 195 countries and territories, 1990–2017: A systematic analysis for the Global Burden of Disease Study 2017

Background: The Global Burden of Diseases, Injuries, and Risk Factors Study 2017 (GBD 2017) includes a comprehensive assessment of incidence, prevalence, and years lived with disability (YLDs) for 354 causes in 195 countries and territories from 1990 to 2017. Previous GBD studies have shown how the decline of mortality rates from 1990 to 2016 has led to an increase in life expectancy, an ageing global population, and an expansion of the non-fatal burden of disease and injury. These studies have also shown how a substantial portion of the world's population experiences non-fatal health loss with considerable heterogeneity among different causes, locations, ages, and sexes. Ongoing objectives of the GBD study include increasing the level of estimation detail, improving analytical strategies, and increasing the amount of high-quality data. Methods: We estimated incidence and prevalence for 354 diseases and injuries and 3484 sequelae. We used an updated and extensive body of literature studies, survey data, surveillance data, inpatient admission records, outpatient visit records, and health insurance claims, and additionally used results from cause of death models to inform estimates using a total of 68 781 data sources. Newly available clinical data from India, Iran, Japan, Jordan, Nepal, China, Brazil, Norway, and Italy were incorporated, as well as updated claims data from the USA and new claims data from Taiwan (province of China) and Singapore. We used DisMod-MR 2.1, a Bayesian meta-regression tool, as the main method of estimation, ensuring consistency between rates of incidence, prevalence, remission, and cause of death for each condition. YLDs were estimated as the product of a prevalence estimate and a disability weight for health states of each mutually exclusive sequela, adjusted for comorbidity. We updated the Socio-demographic Index (SDI), a summary development indicator of income per capita, years of schooling, and total fertility rate. Additionally, we calculated differences between male and female YLDs to identify divergent trends across sexes. GBD 2017 complies with the Guidelines for Accurate and Transparent Health Estimates Reporting. Findings: Globally, for females, the causes with the greatest age-standardised prevalence were oral disorders, headache disorders, and haemoglobinopathies and haemolytic anaemias in both 1990 and 2017. For males, the causes with the greatest age-standardised prevalence were oral disorders, headache disorders, and tuberculosis including latent tuberculosis infection in both 1990 and 2017. In terms of YLDs, low back pain, headache disorders, and dietary iron deficiency were the leading Level 3 causes of YLD counts in 1990, whereas low back pain, headache disorders, and depressive disorders were the leading causes in 2017 for both sexes combined. All-cause age-standardised YLD rates decreased by 3·9% (95% uncertainty interval [UI] 3·1-4·6) from 1990 to 2017; however, the all-age YLD rate increased by 7·2% (6·0-8·4) while the total sum of global YLDs increased from 562 million (421-723) to 853 million (642-1100). The increases for males and females were similar, with increases in all-age YLD rates of 7·9% (6·6-9·2) for males and 6·5% (5·4-7·7) for females. We found significant differences between males and females in terms of age-standardised prevalence estimates for multiple causes. The causes with the greatest relative differences between sexes in 2017 included substance use disorders (3018 cases [95% UI 2782-3252] per 100 000 in males vs 1400 [1279-1524] per 100 000 in females), transport injuries (3322 [3082-3583] vs 2336 [2154-2535]), and self-harm and interpersonal violence (3265 [2943-3630] vs 5643 [5057-6302]). Interpretation: Global all-cause age-standardised YLD rates have improved only slightly over a period spanning nearly three decades. However, the magnitude of the non-fatal disease burden has expanded globally, with increasing numbers of people who have a wide spectrum of conditions. A subset of conditions has remained globally pervasive since 1990, whereas other conditions have displayed more dynamic trends, with different ages, sexes, and geographies across the globe experiencing varying burdens and trends of health loss. This study emphasises how global improvements in premature mortality for select conditions have led to older populations with complex and potentially expensive diseases, yet also highlights global achievements in certain domains of disease and injury

Global, regional, and national age-sex-specific mortality and life expectancy, 1950-2017: a systematic analysis for the Global Burden of Disease Study 2017

Background: Assessments of age-specific mortality and life expectancy have been done by the UN Population Division, Department of Economics and Social Affairs (UNPOP), the United States Census Bureau, WHO, and as part of previous iterations of the Global Burden of Diseases, Injuries, and Risk Factors Study (GBD). Previous iterations of the GBD used population estimates from UNPOP, which were not derived in a way that was internally consistent with the estimates of the numbers of deaths in the GBD. The present iteration of the GBD, GBD 2017, improves on previous assessments and provides timely estimates of the mortality experience of populations globally. Methods: The GBD uses all available data to produce estimates of mortality rates between 1950 and 2017 for 23 age groups, both sexes, and 918 locations, including 195 countries and territories and subnational locations for 16 countries. Data used include vital registration systems, sample registration systems, household surveys (complete birth histories, summary birth histories, sibling histories), censuses (summary birth histories, household deaths), and Demographic Surveillance Sites. In total, this analysis used 8259 data sources. Estimates of the probability of death between birth and the age of 5 years and between ages 15 and 60 years are generated and then input into a model life table system to produce complete life tables for all locations and years. Fatal discontinuities and mortality due to HIV/AIDS are analysed separately and then incorporated into the estimation. We analyse the relationship between age-specific mortality and development status using the Socio-demographic Index, a composite measure based on fertility under the age of 25 years, education, and income. There are four main methodological improvements in GBD 2017 compared with GBD 2016: 622 additional data sources have been incorporated; new estimates of population, generated by the GBD study, are used; statistical methods used in different components of the analysis have been further standardised and improved; and the analysis has been extended backwards in time by two decades to start in 1950. Findings: Globally, 18·7% (95% uncertainty interval 18·4–19·0) of deaths were registered in 1950 and that proportion has been steadily increasing since, with 58·8% (58·2–59·3) of all deaths being registered in 2015. At the global level, between 1950 and 2017, life expectancy increased from 48·1 years (46·5–49·6) to 70·5 years (70·1–70·8) for men and from 52·9 years (51·7–54·0) to 75·6 years (75·3–75·9) for women. Despite this overall progress, there remains substantial variation in life expectancy at birth in 2017, which ranges from 49·1 years (46·5–51·7) for men in the Central African Republic to 87·6 years (86·9–88·1) among women in Singapore. The greatest progress across age groups was for children younger than 5 years; under-5 mortality dropped from 216·0 deaths (196·3–238·1) per 1000 livebirths in 1950 to 38·9 deaths (35·6–42·83) per 1000 livebirths in 2017, with huge reductions across countries. Nevertheless, there were still 5·4 million (5·2–5·6) deaths among children younger than 5 years in the world in 2017. Progress has been less pronounced and more variable for adults, especially for adult males, who had stagnant or increasing mortality rates in several countries. The gap between male and female life expectancy between 1950 and 2017, while relatively stable at the global level, shows distinctive patterns across super-regions and has consistently been the largest in central Europe, eastern Europe, and central Asia, and smallest in south Asia. Performance was also variable across countries and time in observed mortality rates compared with those expected on the basis of development. Interpretation: This analysis of age-sex-specific mortality shows that there are remarkably complex patterns in population mortality across countries. The findings of this study highlight global successes, such as the large decline in under-5 mortality, which reflects significant local, national, and global commitment and investment over several decades. However, they also bring attention to mortality patterns that are a cause for concern, particularly among adult men and, to a lesser extent, women, whose mortality rates have stagnated in many countries over the time period of this study, and in some cases are increasing